Automatic presence simulator for security systems

ABSTRACT

Methods, systems and computing devices are described herein for monitoring a premises when residents or other users are present in order to detect patterns of activity at the premises. Such patterns may comprise, for example, a typical schedule indicating usage of one or more devices by one or more users of the premises. When a user is away or otherwise inactive, commands may be sent to various user devices to make it appear (e.g., to those outside the premises) as if the user is present by simulating and/or recreating the patterns that were previously detected.

BACKGROUND

In providing security services, security systems typically includesensors for detecting break-ins. Such sensors may detect intrusions viawindows and doors while a user is away or while the security system isotherwise armed. However, there remains an ever-present need forimprovements in security systems' ability to detect, and hopefullyprevent, unwanted intrusions.

SUMMARY

The following presents a simplified summary in order to provide a basicunderstanding of the disclosure. The summary is not an extensiveoverview of the disclosure. It is neither intended to identify key orcritical elements of the disclosure nor to delineate the scope of thedisclosure. The following summary merely presents some concepts of thedisclosure in a simplified form as a prelude to the description below.

Methods, systems and computing devices are described herein formonitoring a premises when residents or other users are present in orderto detect patterns of activity at the premises. The patterns may bedetected from recorded audio events indicating, for example, the soundsof television programs being viewed, the turning on/off of electronicdevices, use of appliances, etc. Such patterns may comprise, forexample, a typical schedule indicating usage of one or more devices byone or more users of the premises. When a user is away or otherwiseinactive, commands may be sent to various user devices to make it appear(e.g., to those outside the premises) as if the user is present bysimulating and/or recreating the patterns that were previously detected.

The methods, systems and computing devices described herein may beincluded as part of a network, such as a cable television distributionnetwork.

The details of these and other embodiments of the present disclosure areset forth in the accompanying drawings and the description below. Otherfeatures and advantages will be apparent from the description anddrawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example and not limitedin the accompanying figures in which like reference numerals indicatesimilar elements and in which:

FIG. 1 shows an example network on which the various elements describedherein may be implemented.

FIG. 2 shows an example computing device on which the various elementsdescribed herein may be implemented.

FIG. 3 shows an example monitoring and security system within apremises.

FIGS. 4A-B show an example monitoring and security system operating indifferent modes.

FIG. 5 shows an example premises controller for implementing thefunctions described herein.

FIGS. 6A-B show example data structures for storing events andgenerating user profiles.

FIGS. 7A-B show example user interfaces for adjusting a user profile.

FIG. 8 shows an example user interface for identifying audio signalsreceived by the monitoring and security system.

FIG. 9 shows an example data structure of a user profile.

FIG. 10 shows an example process flow diagram implemented by a processorfor carrying out monitoring and security functions described herein.

FIG. 11 shows an example process flow diagram for verifying execution ofcommands transmitted to user entertainment devices.

DETAILED DESCRIPTION

In the following description of various illustrative embodiments,reference is made to the accompanying drawings, which form a parthereof, and in which is shown, by way of illustration, variousembodiments in which aspects of the disclosure may be practiced. It isto be understood that other embodiments may be utilized, and structuraland functional modifications may be made, without departing from thescope of the present disclosure.

FIG. 1 shows an example network 100 on which many of the variousfeatures described herein may be implemented. The network 100 may be anytype of information distribution network, such as satellite, telephone,cellular, wireless, optical fiber network, coaxial cable network, and/ora hybrid fiber/coax (HFC) distribution network. Additionally, thenetwork 100 may be a combination of networks. The network 100 may use aseries of interconnected communication links 101 (e.g., coaxial cables,optical fibers, wireless, etc.) and/or some other network (e.g., theInternet, a PSTN, etc.) to connect an end-point to a local office orheadend 103. Example end-points are shown in FIG. 1 as a premises 102(e.g., businesses, homes, consumer dwellings, etc.) The local office 103(e.g., a data processing and/or distribution facility) may transmitinformation signals onto the links 101, and each premises 102 may have areceiver used to receive and process those signals.

There may be one link 101 originating from the local office 103, and itmay be split a number of times to distribute the signal to various homes102 in the vicinity (which may be many miles) of the local office 103.The links 101 may include components not shown, such as splitters,filters, amplifiers, etc. to help convey the signal clearly, but ingeneral each split introduces a bit of signal degradation. Portions ofthe links 101 may also be implemented with fiber-optic cable, whileother portions may be implemented with coaxial cable, other links, orwireless communication paths.

The local office 103 may include a termination system (TS) 104, such asa cable modem termination system (CMTS) in a HFC network, a DSLAM in aDSL network, a cellular base station in a cellular network, or someother computing device configured to manage communications betweendevices on the network of links 101 and backend devices such as servers105-107 (which may be physical servers and/or virtual servers, forexample, in a cloud environment). The TS may be as specified in astandard, such as the Data Over Cable Service Interface Specification(DOCSIS) standard, published by Cable Television Laboratories, Inc.(a.k.a. CableLabs), or it may be a similar or modified device instead.The TS may be configured to place data on one or more downstreamfrequencies to be received by modems or other user devices at thevarious premises 102, and to receive upstream communications from thosemodems on one or more upstream frequencies. The local office 103 mayalso include one or more network interfaces 108, which can permit thelocal office 103 to communicate with various other external networks109. These networks 109 may include, for example, networks of Internetdevices, telephone networks, cellular telephone networks, fiber opticnetworks, local wireless networks (e.g., WiMAX), satellite networks, andany other desired network, and the interface 108 may include thecorresponding circuitry needed to communicate on the network 109, and toother devices on the network such as a cellular telephone network andits corresponding cell phones.

As noted above, the local office 103 may include a variety of servers105-107 that may be configured to perform various functions. The serversmay be physical servers and/or virtual servers. For example, the localoffice 103 may include a push notification server 105. The pushnotification server 105 may generate push notifications to deliver dataand/or commands to the various homes 102 in the network (or morespecifically, to the devices in the homes 102 that are configured todetect such notifications). The local office 103 may also include acontent server 106. The content server 106 may be one or more computingdevices that are configured to provide content to users in the homes.This content may be, for example, video on demand movies, televisionprograms, songs, text listings, etc. The content server 106 may includesoftware to validate user identities and entitlements, locate andretrieve requested content, encrypt the content, and initiate delivery(e.g., streaming) of the content to the requesting user and/or device.

The local office 103 may also include one or more application servers107. An application server 107 may be a computing device configured tooffer any desired service, and may run various languages and operatingsystems (e.g., servlets and JSP pages running on Tomcat/MySQL, OSX, BSD,Ubuntu, Redhat, HTML5, JavaScript, AJAX and COMET). For example, anapplication server may be responsible for collecting television programlistings information and generating a data download for electronicprogram guide listings. Another application server may be responsiblefor monitoring user viewing habits and collecting that information foruse in selecting advertisements. Another application server may beresponsible for formatting and inserting advertisements in a videostream being transmitted to the premises 102. Another application servermay be responsible for formatting and providing data for an interactiveservice being transmitted to the premises 102 (e.g., chat messagingservice, etc.). In some examples, an application server may implement anetwork controller 303, as further described with respect to FIG. 3below.

An example premises 102 a may include an interface 120. The interface120 may comprise a modem 110, which may include transmitters andreceivers used to communicate on the links 101 and with the local office103. The modem 110 may be, for example, a coaxial cable modem (forcoaxial cable links 101), a fiber interface node (for fiber optic links101), or any other desired device offering similar functionality. Theinterface 120 may also comprise a gateway interface device 111 orgateway. The modem 110 may be connected to, or be a part of, the gatewayinterface device 111. The gateway interface device 111 may be acomputing device that communicates with the modem 110 to allow one ormore other devices in the premises to communicate with the local office103 and other devices beyond the local office. The gateway 111 maycomprise a set-top box (STB), digital video recorder (DVR), computerserver, or any other desired computing device. The gateway 111 may alsoinclude (not shown) local network interfaces to provide communicationsignals to devices in the premises, such as display devices 112 (e.g.,televisions), additional STBs 113, personal computers 114, laptopcomputers 115, wireless devices 116 (wireless laptops and netbooks,mobile phones, mobile televisions, personal digital assistants (PDA),etc.), a landline phone 117, and any other desired devices. Examples ofthe local network interfaces include Multimedia Over Coax Alliance(MoCA) interfaces, Ethernet interfaces, universal serial bus (USB)interfaces, wireless interfaces (e.g., IEEE 802.11), BLUETOOTH®interfaces (including, for example, BLUETOOTH® LE), ZIGBEE®, and others.The premises 102 a may further include one or more listening devices119, the operation of which will be further described below.

FIG. 2 shows an example computing device on which various elementsdescribed herein can be implemented. The computing device 200 mayinclude one or more processors 201, which may execute instructions of acomputer program to perform any of the features described herein. Theinstructions may be stored in any type of computer-readable medium ormemory, to configure the operation of the processor 201. For example,instructions may be stored in a read-only memory (ROM) 202, randomaccess memory (RAM) 203, removable media 204, such as a Universal SerialBus (USB) drive, compact disk (CD) or digital versatile disk (DVD),floppy disk drive, or any other desired electronic storage medium.Instructions may also be stored in an attached (or internal) hard drive205. The computing device 200 may include one or more output devices,such as a display 206 (or an external television), and may include oneor more output device controllers 207, such as a video processor. Theremay also be one or more user input devices 208, such as a remotecontrol, keyboard, mouse, touch screen, microphone, etc. The computingdevice 200 may also include one or more network interfaces, such asinput/output circuits 209 (such as a network card) to communicate withan external network 210. The network interface may be a wired interface,wireless interface, or a combination of the two. The interface 209 mayinclude a modem (e.g., a cable modem), and the network 210 may includethe communication links and/or networks shown in FIG. 1, or any otherdesired network.

The computing device 200 may include a monitoring and securityapplication 211 that implements one or more security or monitoringfeatures of the present description. The monitoring and securityapplication 211 will be further described below with respect to FIGS. 3and 4.

The FIG. 2 example is an exemplary hardware configuration. Modificationsmay be made to add, remove, combine, divide, etc. components as desired.Additionally, the components shown may be implemented using basiccomputing devices and components, and the same components (e.g., theprocessor 201, storage 202, user interface, etc.) may be used toimplement any of the other computing devices and components describedherein.

FIG. 3 shows an example monitoring and security system 300 forimplementing features described herein. A premises may include apremises controller 301. The premises controller 301 monitors thepremises 302 and simulates the presence of a user or resident of thepremises 302. The premises controller 301 may monitor recorded audiosignals in order to detect audio patterns of normal activities at thepremises. The detected patterns may comprise, for example, indicationsof one or more habits of residents of the premises, for example, that aresident usually watches television in the afternoons, sometimes listensto music in the evenings, and/or other habits indicating usage patternsof media devices. When the resident is away, the premises controller 301may command devices of the premises 302 to simulate the user's presence.For example, the premises controller 301 may turn on the television inthe afternoon and turn on music in the evening to create the appearancethat a resident is at home.

The premises controller 301 located in the premises 302 connects to alocal office 311, which in turn connects via the WAN 314 to the networkcontroller 303. The premises 302 further contains a plurality oflistening devices 305 (e.g., devices that include one or moremicrophones) and/or video cameras 310 for monitoring the premises 302.An alarm panel 304 connects to the premises controller 301.Additionally, the premises controller 301 may control user entertainmentdevices 306, including a television 307 and a stereo 319 viatransmission(s) 316. The premises controller 301 may also include homeautomation functions enabling communication with and control of lights308 and other such devices. Various devices such as the alarm panel 304,the listening devices 305, lights 308, and a video camera 310 may beconnected to the premises controller 301 via a local network 312.

The listening devices 305 may be scattered throughout the premises 302.For example, one or more listening devices 305 may be located in eachroom, or in select rooms, of the premises 302. Each listening device 305may include one or more microphones for receiving/recording audiosignals. The listening devices 305 may periodically transmit thereceived audio signals to the premises controller 301 for purposes ofmonitoring the premises 302. The premises controller 301 may analyze andprocess the monitored audio signals independently or in conjunction withthe network controller 303. The listening devices 305 may send audiosignals to the premises controller 301 using dedicated wires, via localnetwork 312, or in any other manner. A listening device 305 may beintegrated with another device, such as an alarm panel 304.

The alarm panel 304 may control security settings of the monitoring andsecurity system 300. For example, a user may change an arming mode ofthe monitoring and security system 300 via the alarm panel 304 in orderto enable or disable certain security features. Arming modes may includean “away” mode, a “night” mode, and/or a “stay” mode, among others. Thepremises controller 301 may check the modes set at the alarm panel 304in order to determine a mode of the premises controller 301. When a modeindicates a user is present, the premises controller 301 may monitor thepremises 302 to detect patterns of normal activity and behavior. When amode indicates a user is away, the premises controller 301 may simulatethe user's presence at the premises.

A portable communication device 317 (e.g., a smartphone) and/or apersonal computer 318 may connect to the premises 302 via the WAN 313(in conjunction with cellular network 315) and/or the WAN 314. Theportable communication device 317 and/or the personal computer 318 maycommunicate with the network controller 303, which may in turn relaycommunications to and from the premises controller 301. Suchcommunications may include requesting information from the securitysystem, modifying a setting, or the like. For example, a resident couldmodify a user profile generated by the premises controller 301 in orderto determine what actions the premises controller 301 takes in theuser's absence from the premises 302.

Additionally or alternatively, the portable communication device 317and/or the personal computer 318 may communicate with the premisescontroller 301 without the involvement of the network controller 303.The network controller 303 may perform the functions ascribed herein tothe premises controller 301 instead of or in addition to the premisescontroller 301. The network controller 303 may be integrated with thelocal office 311 (e.g., as an application server 107 as shown by FIG.1). Accordingly, an application server 107 embodying the networkcontroller 303 may perform any of the techniques described herein.

The premises controller 301 may be implemented as a hardware or softwarecomponent of the computing device 200 (e.g., as a monitoring andsecurity application 211). Additionally or alternatively, the premisescontroller 301 may be implemented as a standalone device.

FIG. 4A shows the monitoring and security system operating in a firstmode in which a user 320 is at the premises 302 (e.g., a stay mode). Theuser 320 may interact with one or more user entertainment devices 306,such as the television 307 and the stereo 319 (e.g., using remotecontrols or other user interfaces). The user entertainment devices 306may generate sound. Additionally, the user 320 may generate sound, forexample, by speaking. The listening devices 305 may record the soundsgenerated in the premises 302 and send the recorded audio signals to thepremises controller 301 for analysis.

The premises controller 301 may analyze each audio signal to determineor estimate meta information about the audio contained in the signal,such as a volume of the audio signal, a classification of the audiosignal (e.g., speech, music, etc.), and/or a device or person which mayhave generated the audio signal. The recorded audio signal together withthe meta information may be stored as an event in an event log. Thepremises controller 301 may further analyze the event log to determineinformation about patterns of activity.

The premises controller 301 may analyze several days' worth of loggedevents in order to determine a typical day for a user 320. After findingpatterns characterizing the typical day, the premises controller 301 maycreate a schedule of commands that may be sent to the user entertainmentdevices 306 to simulate the presence of the user 320.

Additionally, the premises controller 301 may analyze the event logs todetermine pairs of events. For example, the premises controller 301 maydetermine that a listening device 305 frequently records a dog barkingafter a doorbell rings. The premises controller 301 may create a dynamicevent that can be triggered by the first event of the pair. Accordingly,when a user is away and the premises controller 301 detects a doorbellringing, it may instruct a stereo to play the sound of a dog barking.Accordingly, a user profile may be updated with dynamic events thatfurther characterize patterns of activity for the premises 302.

FIG. 4B shows the monitoring and security system operating in a secondmode in which a user 320 is away from the premises 302 (e.g., an awaymode). The premises controller 301 may send commands to the userentertainment device 306 at certain times as instructed by the scheduleof commands. Additionally, the listening devices 305 may continuemonitoring the premises for dynamic event triggers. When such a triggerevent is detected, an appropriate command may be sent to a userentertainment device 306 in response to the dynamic event.

FIG. 5 shows an example hardware embodiment of the premises controller301. The premises controller 301 contains a processor 501 comprisingseveral modules 502-507 for implementing functions described herein. Theprocessor 501 stores data and accesses data at the storage 508. Theprocessor 501 also transmits (and/or receives) transmissions fromtransceivers 510 a-n. The processor 501 connects to a local networkinterface 509, which may further connect (via a local network 312) to agateway 111. Signals 511 may be received at the processor frommonitoring devices (e.g., listening devices 305 and/or video cameras310, etc.).

The processor 501 includes a command generator 502 that generatescommands or sequences of commands for transmission to user entertainmentdevices 306 via transceivers 510 a-n. The command generator 502 mayverify that the commands were successfully executed by listening toaudio signals received by listening devices 305. For example, aftersending a command to a television via an infrared signal, the commandgenerator 502 may check signals 511 to ensure that appropriate soundsare being generated by the television. Accordingly, the commandgenerator 502 may be configured to execute sub-routines to verify that acommand or commands were successfully executed by the target userentertainment device 306. The audio analyzer 503 analyzes audio signalsreceived from listening devices 305 in order to detect and/or log audioevents. The audio signals may capture sounds generated by userentertainment devices 306, users or residents of the premises 302 (e.g.,speech), other devices (e.g., appliances) of the premises 302, or anyother sources of audio at the premises 302. The audio analyzer 503 mayanalyze the audio signals in order to determine meta information aboutthe audio signals. The audio analyzer 503 may determine or estimate avolume of the audio signal, a classification of the audio signal, a roomof the audio signal, a device or person that generated the audio signal,as well as other properties.

Similarly, the video analyzer 505 may analyze video signals receivedfrom the video cameras 310 in order to detect and log video events. Forexample, the video analyzer 505 may analyze video signals to determinewhen lights are on or off. The signals 511 may be received fromdedicated interface(s) of the premises controller 301, via one or moreof transceivers 510 a-n, and/or via the local network interface 509.Signals 511 may be received periodically or continuously from thelistening devices 305 and/or video cameras 310.

The profile Generator 504 may generate one or more user profiles 512indicating typical behavior for one or more users. A user profile 512may be generated based on a log 515 of events analyzed by the audioanalyzer 503, video analyzer 505, and/or automation controller 507. Forexample, if the user typically watches a news program from 7 am to 8 am,listens to music on a stereo from 1 pm to 3 pm, and has the officelights turned on from 7 am to 9 pm, then this behavior may be detectedby the audio analyzer 503 and video analyzer 505, and information may bestored in the user profile to indicate this behavior. The user profile512 may also include a schedule of commands that can be sent to userentertainment devices and/or other controlled devices to create theappearance that one or more residents is at home by, for example,indicating a predetermined time (and/or a randomized time with respectto a predetermined time) when a video device should be activated, and asetting or service that should be activated (e.g., turning on atelevision and tuning to the news program from 7 am to 8 am, tuning to astereo should from 1 pm to 3 pm, and turning on the office lights from 7am to 9 pm). When the system determines that the house is unattended,the system may send automatic commands to control thetelevision/stereo/lights to mimic the same behavior.

The security monitor 506 may implement security functions such assetting and triggering alarms, detecting break-ins, etc. The securitymonitor 506 may further send alerts or messages to remote devices suchas the portable communication device 317 and/or personal computer 318.The automation controller 507 may implement home automation protocolsfor controlling lights and other smart devices via one or more homeautomation protocols (e.g., X10, Z-WAVE®, INSTEON®, ZIGBEE®, and othersuch protocols). The automation controller 507 may also monitor signalstransmitted to and/or received from smart devices indicating events(e.g., lights being turning on or off) and log such events in the log515.

Each of the modules 502-507 may be implemented as software modules,components of software programs, combinations of hardware and/orsoftware components, or the like.

All or a portion of the monitoring and security application 211 may beimplemented by the processor 501 and any or all of the modules 502-507.Any or all of the modules 502-507 may be implemented at the networkcontroller 303 instead of or in addition to at the premises controller301.

Storage 508 includes one or more user profiles 512 generated by theprofile generator 504, one or more device profiles 513 specifyingcontrolled devices and their capabilities, one or more source maps 514for mapping media content to particular media sources, a log 515 ofevents detected from monitoring device signals 511, media samples 516for detecting and playing back events, and other data needed by thepremises controller 301. Storage 508 may include any type of storagedevice(s), such as hard drives, flash memory, ROM, RAM, and the like.

The device profiles 513 may indicate device-specific information such aswhat protocol a user entertainment device 306 uses to receive commands,how commands should be formatted, and the like. The device profile 513may further specify a room where the device is located. The audioanalyzer 503 may determine or estimate, based on information indicatingthe room, which device (or devices) may have generated a sound recordedby a listening device 305. Additionally, the premises controller 301 mayselect, based on information indicating the room, which userentertainment device 306 it should command. For example, if a userprofile indicates that a resident frequently talks on the phone in theliving room, the premises controller 301 may simulate that behavior inthe resident's absence by playing an audio file containing speech on theliving room stereo. In this example, the device profile 513 for thestereo indicates it is in the living room.

Source maps 514 may indicate sources of content (e.g., televisionchannels) corresponding to remote control commands for a particularservice. For example, when a premises controller 301 executes a commandto tune a television to MSNBC, it may access the source map 514 todetermine the appropriate channel number to send to the television orset-top box. The source map 514 may further indicate which sources arefree to access. In some examples, the premises controller 301 selectssources from the subset of sources that are free to access in order toavoid accruing charges for simulating a user's presence in the premises302.

Media samples 516 may include media files pre-installed in the storage508, uploaded by a user, and/or downloaded from the network. The mediasamples 516 may be used by the audio analyzer 503 to recognize certainsounds. For example, the user may upload a recording of the doorbell sothat the audio analyzer 503 can recognize the user's doorbell andclassify a corresponding event appropriately. Accordingly, the audioanalyzer 503 can compare media samples 516 to audio signals receivedfrom listening devices 305 to determine metadata about the audiosignals. In this example, the media sample 516 may be tagged (e.g., bythe user 320) with metadata such as a classification of “doorbell,” aparticular room, and/or other such metadata.

Additionally, the media samples 516 may be used by the premisescontroller 301 to play certain sounds. For example, a recording of a dogbarking could be played on the stereo to simulate the presence of a dogwhen a doorbell event is detected. Accordingly, the premises controller301 may download a media sample 516 associated with the “dog barking”classification when a profile specifies such a dynamic event.Additionally or alternatively, the user 320 may manually upload such amedia sample 516, or such a media sample 516 may come pre-installed inthe premises controller 301.

Transceivers 510 a-n may include one or more transmitters and/orreceivers for communicating with user entertainment devices 306 andother devices such as lights 308. For example, a first transceiver 510 amay implement a BLUETOOTH® protocol. A transmitter 510 b may be aninfrared (IR) blaster for emulating remote control commands, such ascommands for a television 307 or stereo 319. A transceiver 510 c mayimplement a radio frequency communication protocol, such as ZIGBEE® orWI-FI®. Other transceivers 510 n could be used as well.

FIG. 6A shows example data structures including example logs 601 a-b,groups 602 a-c of events, and habits 603 a-b. Logs 601 may include aplurality of events including meta information about each event. Suchmeta information may include, as shown in FIG. 6A, time information,classification information, volume information, and room information.The meta information may be determined by the audio analyzer 503 andstored in the log 601 for analysis.

The profile generator 504 may analyze the logs 601 and generate data forconstructing a user profile. The profile generator 504 may perform atwo-step process for generating groups 602 of events from logs 601,followed by generating habits 603 from groups 602 of events.Additionally or alternatively, the profile generator 504 may use otherprocesses to analyze the log 601 data and generate a profile therefrom.

FIG. 6A shows a portion of a log 601 a for a first day and a portion ofa log 601 b for a second day. The profile generator 504 may group eventsby common values of meta information. The profile generator 504 maygroup events by time and room information in order to generate groups602 of events. For example, the profile generator 504 may determine thatmultiple events in a first log 601 a occur in a time period of 1:10-1:14in the living room. The profile generator 504 may generate a group 602 acorresponding to those events.

Such groups 602 may be formed of events having relatively contiguoustime values for a given room. For example, if a log 601 a containsanother event entry at time 1:46 in the living room, the profilegenerator 504 may avoid adding such an event to the first group 602 abecause 1:46 is too far in time from the remaining members of the group.Events over a threshold amount of time (e.g., 30 minutes) differencefrom other events may be split into different groups 602.

In the example of FIG. 6A, the profile generator 504 may analyze theevents corresponding to the group 602 a and determine that the majorityof such events have a “speech” classification. Therefore, the profilegenerator 504 may assign a “speech” class to the group 602 a. Theprofile generator 504 may detect that above a threshold percentage ofevents in a group 602 have a single class value, and assign that classvalue to the group 602. For example, if more than 75% of events in agroup 602 have a “speech” value, the entire group may be assigned a“speech” value.

The profile generator 504 may further determine that all of the eventsin the example group 602 a (or a subset of events, for example thesubset of events classified as “speech”) have volume properties within agiven range (e.g., 60-65 dB). Accordingly, the profile generator 504 mayassign the given volume range to the group 602 a as shown in FIG. 6A.

The profile generator 504 may determine multiple groups 602 from asingle log 601 a. For example, the profile generator 504 may determine asecond group 602 b corresponding to events occurring between 1:10-1:16in the basement. The profile generator 504 may assign a classificationand a volume range to the group 602 b in a similar manner as for thegroup 602 a.

The profile generator 504 may determine additional groups from a log 601b for a second day. The profile generator 504 may be configured toassign multiple classes to a single group 602, such as when events ofthe group 602 are split fairly evenly between classifications. Forexample, because the log 601 b contains a number of events with bothmusic and speech classifications, the group 602 c may be assigned twoclassifications as shown in FIG. 6A. The remaining properties of thegroup 602 c may be assigned as for the other groups 602 a, 602 b.

The profile generator 504 may form groups 602 based on one or more otherproperties instead of or in addition to time and room properties. Forexample, the profile generator 504 may form groups 602 of events havingthe same classification, such that each group has at most oneclassification. Additionally, the profile generator 504 may form groups602 based on how close in volume events are to each other, such thatrelatively loud events may be split into different groups fromrelatively quiet events. Additionally, any other properties included asmeta information for an event may be used for forming groups 602. Suchproperties may include a device or person that generated the sound, atype of the event, a frequency property of the event, or any otherproperty of the event.

After forming the groups 602 of events, profile generator 504 mayfurther analyze the groups 602 to form habits 603. Habits 603 may beformed by comparing groups 602 across different days. Groups 602corresponding to two or more days may be compared. When profilegenerator finds matching (or partially matching) groups of events acrossmultiple days, it may create a habit 603. For example, when profilegenerator 504 determines that groups 602 a and 602 c are partial matchesin terms of class (e.g., both groups contain a “speech” class), time ofday (e.g., the groups time ranges are relatively close together), androom (e.g., both groups indicate the living room), it may generate ahabit 603 a based on the properties of groups 602 a and 602 c.

The profile generator 504 may determine a partial match based on one ormore values of meta information being the same or similar. In someexamples, the profile generator 504 may generate a habit if the timeinformation is similar (e.g., if time ranges for the two groups overlapat least partially, or have less than a certain amount of time betweenan end time of one group and a start time of the other, or if some othermeasure of time similarity indicates the time information is similar).In the example of FIG. 6A, the profile generator 504 may determine thatgroups 602 a and 602 c are similar in terms of time because they occurwithin a threshold time of each other (e.g., 2 hours). Other thresholdsmay be used to determine similarity of other properties (e.g., within 10dB for volume).

The profile generator 504 may determine that two groups are a partialmatch even when some properties are not the same or similar. Forexample, two groups 602 b and 602 c may be determined as partial matcheseven though the room values are different.

The profile generator 504 may generate the habits 603 based on theproperties of the matched groups 602. For example, the time range of ahabit 603 a may be set using the earliest and latest times ofcorresponding groups 602 a and 602 c, the volume range may be set usingthe quietest and loudest volumes of corresponding groups 602 a and 602c, etc. The profile generator 504 may determine the properties of ahabit 603 based on the union/overlap of the property values of thematched groups, the average values of the properties of the matchedgroups, or based on some other method of combining the properties of twoor more groups. For example, the value of the room property for a habit603 b may include the values of both room properties for matching groups602 b and 602 c.

The profile generator 504 may use the habits 603 in order to generatecommands for simulating a user's presence. Such commands may begenerated taking into account the devices connected to the system and auser's preferences. The generated commands may be added to a schedule ofthe user profile 512.

FIG. 6B further shows a pair 604 of events detected from multiple days'logs 601 a, 601 b. In the example of FIG. 6B, the profile generator 504detects that events classified as “doorbell” and “dog” frequently occurtogether in a particular order. Accordingly, the profile generator 504may detect that every time (or above a threshold percentage of the time)a first event occurs, a second event occurs. The profile generator 504may create a pair 604 including the first and second events. The pair604 may be used to create dynamic events so that a premises controller301 can dynamically simulate a user's presence.

FIG. 7A shows an example user interface 700 by which a user 320 canadjust a schedule of commands generated by the profile generator 504based on the habits 603. The profile generator 504 may generate commandsbased on properties assigned to the habits 603. For example, commandsfor turning the television on and off may be generated based on a habit603 a.

The profile generator 504 may select a device 702 to receive the one ormore commands based on properties of the habit 603. Command preferencesmay specify mappings between properties of a habit 603 and a device 702that will receive a command. For example, command preferences may map aliving room television to a class property indicating “speech” for thehabit 603 a. The command preferences may indicate that “music” habitsmap to a stereo device. The command preferences may further indicatethat other properties (e.g., a property indicating a room) map toparticular devices 702.

The profile generator 504 may choose the selected device 702 from amongseveral devices indicated by the command preferences based on a roomproperty. For example, the profile generator 504 may select the livingroom television as indicated by the room property of habit 603 a.

After choosing a device 702, the profile generator 504 may construct oneor more commands for the device 702 based on additional properties ofthe habit 603, one or more device profiles 513, a source map 514, andadditional command preferences. For example, the profile generator 504may select “turn on” and “turn off” actions 701 based on the deviceprofile 513 for the living room television indicating it accepts thosecommands. In this example, the profile generator 504 may select “NBC”for an optional source parameter 703 based on a source map 514 andcommand preferences. For example, command preferences may indicate thatNBC should be selected to emulate a speech habit. The profile generator504 may further select times 704 for the commands based on timeproperties of a habit 603.

Similarly, the profile generator 504 may select a stereo device 702 thatmatches a class property indicating “music” for a habit 603 b. Theremaining actions 701, optional parameters 703, and time 704 for thecorresponding commands may be generated by profile generator 504 asdiscussed above. Particularly, actions 701 may be determined based inpart on capabilities of a device that will receive the command. Suchcapabilities may include particular commands or types of commands that adevice can execute. The profile generator 504 may retrieve thecapabilities of a device from a corresponding device profile 513.Optional parameters 703 may be determined based on command preferencesindicating, for example, one or more preferred sources of media content.Time 704 may be determined based on a time property of the habit 603(which may reflect, for example, one or more times associated with aplurality of logged events, or one or more times associated withpatterns of events).

The user interface 700 may be an interactive user interface by which auser 320 can edit the scheduled commands. For example, a user 320 mayselect and/or interact with an element 701-704 of one of the commands inorder to adjust it. Therefore, a user 320 may modify the schedule ofcommands as desired.

FIG. 7B depicts a user interface 705 by which a user 320 may view andmodify a list of dynamic events generated by the profile generator 504.The dynamic events may be generated based on a pair 604 of eventsdetected by the profile generator 504. The profile generator 504 mayselect the first event of the pair 604 (e.g., a doorbell ringing) as thetrigger and create a command that will be executed when the first eventis detected (e.g., playback of a sound of a dog barking).

As before, the profile generator 504 may select a device 702 to receiveone or more commands based on properties of a pair 604 of events. Forexample, to emulate a dog barking, the profile generator 504 may play acorresponding sound file on the stereo. The commands may be configuredbased on command preferences, device profiles 513, and any othersettings.

The user 320 may interact with the interface 705 to edit the list ofdynamic events. The user may add a dynamic event by selecting a trigger707, optionally uploading 708 a sound file corresponding to the trigger,and/or setting one or more of actions 701, device 702, and parameters703 for the command. The profile generator 504 may create a dynamicevent that triggers when the selected trigger 707 is detected (which maybe based on detecting a match with the sound file 708). When the trigger707 is detected, the corresponding command may be executed.

Users 320 may access the user interfaces 700, 705 via a user device,such as a portable communication device 317 or personal computer 318.The user interfaces 700, 705 may be generated by the premises controller301 and/or network controller 303.

FIG. 8 shows a user interface 800 by which a user 320 may identify anevent that the audio analyzer 503 was unable to classify. When the audioanalyzer 503 is unable to classify an event, it may send an alert to auser device, such as the portable communication device 317 or personalcomputer 318. A user of the user device may access the user interface800 to resolve the unidentified event.

The user interface 800 may include a button 801 for playing back theunidentified event. The premises controller 301 may store a recording ofthe unidentified event in storage 508 so that a user can play back therecording upon selection of the user interface playback button 801.

The user interface 800 may further include a selector 802 for selectinga classification of the unidentified event after the user listens to therecording. For example, the audio analyzer 503 may have indicated one ormore classifications with low confidence. Accordingly, the selector 802may contain the indicated classifications indicated along withconfidence levels determined by the audio analyzer 503. Theclassifications may be ranked in order of their determined confidencelevels. Additionally or alternatively, the selector 802 may present alist of all classifications used by the system for selection by theuser.

The selector 802 may further include an additional “not sure” option forindicating that the user cannot identify the sound. The selector 802 mayadditionally include an “ignore” option that causes the sound (andmatching sounds) to be ignored by audio analyzer 503 (e.g., not loggedand/or not simulated) in the future.

The user interface 800 may include additional selectors for indicatingother properties. For example, a selector 803 may be used to indicate adevice from which the sound originated. The list of devices in theselector 803 may be limited to devices in the same room as the listeningdevice 305 that recorded the sound.

After a user indicates the correct values for the event, a log 601 maybe updated with the resolved event data. The profile generator 504 mayre-analyze the updated log 601 after unidentified events have beenresolved by the user.

FIG. 9 shows a data structure for the user profile 512. The user profile512 may comprise a command schedule 901 for storing the schedule ofcommands generated by the profile generator 504, dynamic events 902 forstoring the dynamic events generated by profile generator 504, privacyoptions 903 for controlling listening devices 305, and commandpreferences 904 used to generate the schedule of commands and dynamicevents.

The command schedule 901 and dynamic events 902 may store commands foroperating user entertainment devices 306. The command schedule 901 anddynamic events 902 may also store additional data about commands, suchas which transceivers 510 to use to send a particular command, theformat and/or protocol of commands, and one or more scripts forverifying that a command was successfully executed. The profilegenerator 504 may select a transceiver 510 based on the capabilities ofa device 702 of the command. The profile generator 504 may further storea format and/or protocol for the command as indicated by the deviceprofile 513 for the device 702. For example, a television 307 may beconfigured to accept commands via an IR blaster 510 b. Accordingly, theprofile generator 504 may store (e.g., along with a command for thetelevision 307) an indication of what data to send via the IR blaster510 b in order to execute a command. Additionally, one or more scriptsmay be provided that verify a command was executed. An example script isdescribed below with respect to FIG. 11.

Privacy options 903 may indicate times and/or modes for which selected(or all) listening devices 305 should be disabled. For example, privacyoptions may indicate that all listening devices 305 should be disabledin a night mode, or between certain hours, or that only particularlistening devices 305 should be disabled in certain modes or times. Theaudio analyzer 503 may check the privacy options 903 in order to ignoreaudio signals received from particular listening devices 305 at timeand/or modes specified by the privacy options 903.

Command preferences 904 may indicate preferred settings for constructingcommands by profile generator 504. For example, a user may prefercertain television or audio sources (which may be specified asparameters 703) for emulating habits 603, certain devices 702 foremulating habits 603, and the like. Accordingly, the command preferences904 may contain mappings that indicate how certain properties of habits603 indicate parameters 701-703 for commands. The command preferences904 may additionally or alternatively contain default mappings that auser 320 can override. For example, a default mapping may indicate thata news channel may be tuned on a television to emulate a habit with a“speech” class. However, a user may override the default preference andindicate, for example, that a sports channel should be tuned on thetelevision to emulate the habit with a “speech” class.

FIG. 10 shows an example process flow for processor 501 of premisescontroller 301. The premises controller 301 may be turned on at step1001. Before entering a main loop, the premises controller 301 mayexecute configuration steps 1002, 1003. At step 1002, the premisescontroller 301 may configure one or more user profiles 512. On a firstboot, the premises controller 301 may create a default user profile 512and/or generate configuration prompts for creating one or more userprofiles 512 based on inputs from user(s) 320. A user may be promptedfor favorite sources that may be stored in command preferences 904 ofthe user profile 512. A user may be presented with a user interface 700for configuring a default command schedule 901 of the user profile 512.

At step 1003, the premises controller 301 may configure one or moredevices, such as listening devices 305, user entertainment devices 306,or other premises devices. The premises controller 301 may establishcommunications with such devices in order to determine devicecapabilities and populate a device profile 513 for each device.Additionally or alternatively, the premises controller 301 may promptthe user 320 for information about devices in order to configure deviceprofiles 513. Steps 1002 and/or 1003 may be executed at any time basedon receiving a communication from a user device, such as a portablecommunications device 317 or personal computer 318. For example, a user320 may configure a user profile 512 at any time by connecting to thepremises controller 301 via the Internet.

At step 1003, the premises controller 301 may determine the mode of thesecurity system. The premises controller 301 may regularly poll thealarm panel 304 of the security system to determine the mode, or thealarm panel 304 may send a message to the premises controller 301 whenthe mode is changed automatically or by a user 320. When the mode is setto “stay,” the premises controller 301 may proceed to decision 1005 todetermine whether an event is detected.

At decision 1005, when the mode is set to “stay,” the premisescontroller 301 may monitor events to determine whether an event has beendetected from audio received by listening devices 305 and/or videoreceived from a video camera 310. The audio analyzer 503 of premisescontroller 301 may continually receive and process audio signals fromlistening devices 305 to detect events. For example, the audio analyzer503 may store the most recent N seconds or minutes of audio from eachlistening device 305 (e.g., in a circular buffer). The audio analyzer503 may periodically or continually receive and analyze portions of anyor all of the audio signals in order to detect events. The audioanalyzer 503 may measure temporal characteristics of the audio signals,such as volume over time or change in volume over time, in order todetect an event. Thus loud noises recorded in the audio signals may beflagged as events. Accordingly, the audio analyzer 503 may indicate anevent when the volume of one or more audio signals is above a threshold(e.g., 55 dB), remains above a threshold for a certain duration of time,and/or changes in volume at certain rates.

The audio analyzer 503 may additionally or alternatively analyzefrequency characteristics of the audio signal, such as frequencycomponents, spectral centroid, spectral flux, and/or spectral density todetect events. The audio analyzer 503 may flag signals containingcertain frequency characteristics as events. The audio analyzer 503 mayuse Fourier transforms, wavelet transforms, and the like to obtain thefrequency characteristics of the signal to detect an event.

Upon detecting an event, the audio analyzer 503 may store an indicationof an event along with the audio corresponding to the detected event.Accordingly, when an event is detected at decision 1005, the premisescontroller may analyze the event to determine its properties at step1006.

The premises controller 301, via the audio analyzer 503, analyzes eventsin order to determine information about the event that can be stored ina log 601 for later analysis by the profile generator 504. The audioanalyzer 503 may analyze features of the audio signal in order todetermine properties of the event. The audio analyzer 503 may use one ormore classification algorithms to analyze the audio signalscorresponding to each event. Accordingly, the audio analyzer 503 mayextract one or more features from the audio signal and classify thedetected event using one or more classification algorithms. Suchfeatures may include time-domain features (such as short-time energy,average zero crossing rate, etc.), frequency-domain features (such asmel frequency cepstral coefficients, spectral flux, spectral centroid,etc.), or other features of the audio signal. The classificationalgorithms may determine whether a detected event is recognized by aclassifier trained to categorize audio signals as matching one or moreof a defined set of classes (e.g., speech, music, a dog barking, etc.).Various classification algorithms, such as linear classifiers, supportvector machines, neural networks, and other such classificationalgorithms can be used by the audio analyzer 503 to analyze the eventsat step 1006.

The audio analyzer 503 may additionally detect a volume property of anevent and assign a corresponding volume property. The audio analyzer 503may calculate an average volume of the event, a max volume of the event,or some other measurement of volume. The audio analyzer 503 mayoptionally use a frequency weighting filter such as A-weighting, orITU-R 468 noise weighting, in order to more realistically measureperceived volume. Thus audio analyzer 503 may further assign a volumeproperty as part of step 1006.

The audio analyzer 503 may additionally determine or estimate a roomassociated with the event and assign a corresponding room property. Theaudio analyzer 503 may determine or estimate the room based on whichlistening device 305 recorded the audio signal corresponding to theevent. The audio analyzer 503 may look up the room associated with thelistening device 305 that recorded the corresponding audio signal (e.g.,from a device profile 513 for the listening device 305), and assign theroom property accordingly. Multiple listening devices 305 may generatesimilar events at the same time (e.g., because multiple listeningdevices recorded the same event). The audio analyzer 503 may detect thatsimilar audio signals were recorded by multiple listening devices 305 bycomparing the audio signals, or the properties of events detected fromthe audio signals. Accordingly, the audio analyzer 503 can estimate aroom or rooms where the event may have occurred based on the volumeand/or timing of each received audio signal (e.g., the highest volumeand/or earliest received audio signal may indicate the closest listeningdevice 305). Additionally, the audio analyzer 503 may use relativetiming to compare the audio signals to “triangulate” a distance and/orlocation of the audio event relative to the listening devices 305 inorder to determine a room. Each listening device 305 may containmultiple microphones (e.g., an array of directional microphones)generating multiple audio signals in order to assist in room estimation.Such listening devices 305 could process the multiple audio signalslocally in order to estimate a direction and/or location of the audioevent before transmitting that information to the audio analyzer 503.Accordingly, the audio analyzer 503 may assign room properties as partof step 1006.

Other properties may be assigned by the audio analyzer 503. For example,the one or more classifiers may further indicate sub-classifications,such as a genre of a music class, a gender or age for a speech class andthe like. The audio analyzer 503 may identify a particular piece ofmedia content, for example, by comparing a fingerprint of the audiosignal to a database of media content fingerprints. Such a comparisoncould be used to identify additional meta information about the audiosignal. The audio analyzer 503 may also estimate a device that generatedan event based on the room and class properties. For example, if anevent is classified as “music” and has a room property indication“living room,” the audio analyzer 503 may estimate that a living roomstereo generated the event and assign a corresponding device property.

At step 1007, the premises controller 301 stores the analyzed event(e.g., indicators of the event and its associated properties determinedby the audio analyzer 503) in a log 601. The log 601 may additionallycontain a timestamp marking the time when an event was recorded by alistening device 305. Thus, the premises controller 301 may build up thelog 601 over time so that it can be processed by the profile generator504.

The audio analyzer 503 may attempt to classify an audio signalcorresponding to an event, but classification algorithms may indicate alow confidence of an accurate classification. In that case, the eventmay be classified as “unidentified” in the log 601. At decision 1008,the premises controller 301 may check whether the log 601 contains anevent classified as “unidentified.” When such an event is detected, thepremises controller 301 may send an alert to a user device so the user320 can identify the event if desired. The alert may contain a link bywhich a user can access a user interface 800 for resolving unidentifiedevents. The user identification of the event can be used to re-train orotherwise improve classifiers.

At decision 1010, the premises controller 301 may determine whether toupdate a user profile. The premises controller 301 may cause the profilegenerator 504 to update the user profile 512 at regular intervals, e.g.,daily or weekly. Additionally or alternatively, the profile generator504 may update the user profile 512 when a log 601 contains above athreshold amount of data, above a threshold number of events, or someother measure indicating an amount of data in the log 601. The profilegenerator 504 may update the user profile 512 regardless of the alarmmode. When the profile generator 504 determines to update the userprofile 512, it processes data in the log 601 to determine a commandschedule 901 and/or dynamic events 902.

At step 1011, the profile generator 504 creates groups 602 of eventshaving common properties (e.g., events that occur in common rooms withincommon time frames). The profile generator 504 thus summarizes what washappening within the premises 302 at given times. The groups 602 ofevents may be assigned a particular class common to the events withinthe group 602 (or above a threshold number or percentage of the eventswithin the group 602). For example, a group may be assigned a “speech”class because a majority of the events within the group were classifiedas speech events.

The groups 602 of events may be assigned a volume property in additionto or as an alternative to the class property. The assigned volumeproperty may be an average of the volume properties assigned to theevents of the group 602, or it may be a range encompassing all (or amajority) of the volume properties of the events of the group 602.

At step 1012, the profile generator 504 may compare two or more eventgroups 602 across multiple days in order to determine patterns ofactivity at the premises. The profile generator 504 generates habits 603that indicate such patterns of activity based on the group comparison(e.g., by combining properties of the two or more groups 602, averagingsuch properties, finding the union of such properties, etc.).

At step 1013, the profile generator 504 may map the habits 603 to one ormore commands that emulate the habits 603 when a user is away. Forexample, a habit 603 indicating a user often listens to music in theafternoon may map to a command to turn on a user's stereo in the earlyafternoon and turn it off in the late afternoon. The mappings of habitsto commands may be based on one or more command preferences 904. Thecommand preferences 904 may indicate what types of commands should begenerated based on properties of habits 603. For example, a habit with a“speech” property may be mapped to a particular device (e.g., atelevision) and source (e.g., MSNBC), so that the profile generator 504creates appropriate commands to operate the device at a time indicatedby the habit 603.

At step 1014, the profile generator 504 searches for pairs 604 of eventsin the logs 601. Pairs 604 may include one or more events thatfrequently occur together in a particular order. Examples of such pairs604 may be “phone ringing” followed by “speech,” “doorbell ringing”followed by “dog barking,” “music” followed by “dim lights,” and thelike. The profile generator 504 may find the pairs 604 in the log 601 bydetermining that above a threshold percentage of the time a first eventis detected, a second event is detected within a particular time frame(e.g., 30 seconds). Additionally or alternatively, the profile generator504 may find such pairs 604 based on determining that two events occurin a particular order (e.g., in a particular time frame) above athreshold number of times.

At step 1015, the profile generator 504 generates dynamic events 902based on the pairs of events. The first event of the pair may be set asa trigger 706 for sending a command when an event corresponding to thetrigger 706 is detected. For example, a command may be generated to playan audio recording of speech after premises controller 301 detects aphone ringing event. The profile generator 504 may use commandpreferences 904 in a similar fashion as for step 1013.

At step 1016, the profile generator 504 updates the command schedule 901of the user profile 512 with the commands generated at step 1013, andupdates the dynamic events 902 of the user profile 512 with the triggersand commands generated at step 1015. The update of the profile 512 maythen be completed and the process flow may return to decision 1004 andrestart the “stay” mode loop.

When the mode is set to “away,” the premises controller 301 attempts toemulate patterns of behavior at the residence using the command schedule901 and dynamic events 902 of the user profile 512. Accordingly, whenthe premises controller 301 detects an away mode at decision 1004, thepremises controller 301 may proceed to decision 1020. At decision 1020,the premises controller 301 may compare the current time to the commandschedule to determine whether any scheduled commands are due forexecution.

When a current time matches a time in the command schedule, at step 1022the premises controller 301 causes the command generator 502 to transmita corresponding command (or commands) to one or more user entertainmentdevices or other devices in accordance with the command schedule 901. Asingle entry in the command schedule may contain multiple commands forone or more devices. For example, a scheduled entry for tuning atelevision to a certain channel may include multiple commands forturning on a television, changing the volume of the television, changingthe channel of a set top box, etc. In addition to or as an alternativeto sending a command at an exact time indicated by the command schedule901, the command generator 502 may transmit the command at a randomizedtime with respect to the time indicated in the command schedule 901. Forexample, a command scheduled for 7:00 PM may be transmitted at a randomtime within 15 minutes before and/or after 7:00 PM.

At step 1023, the command generator 502 may optionally verify that thecommand (or commands) was successfully executed. The command generator502 may execute one or more sub-routines to verify that an action wassuccessfully performed by the instructed user entertainment device basedon feedback from listening devices 305. For example, the listeningdevices 305 may record an audio signal containing music after thecommand generator 502 sends a command to play music, which may indicatethat the command was successfully executed. An example of such asub-routine is further described with respect to FIG. 11.

At decision 1021, the premises controller 301 monitors listening devices305 (e.g., via audio analyzer 503) to detect an event that matches adynamic event trigger 706. The audio analyzer 503 may detect events in asimilar manner as for decision 1005 and analyze the event in a similarmanner as for step 1006. The audio analyzer 503 may use the sameclassifiers as for step 1006 to determine properties of the event beforematching the event to a dynamic event trigger 706. The audio analyzer503 may use different classifiers in addition to or instead of theclassifiers described with respect to step 1006. For example, the audioanalyzer 503 may use real-time classifiers to detect and/or analyzeevents at decision 1021. Real-time classifiers may be optimized toquickly detect short-term events such as a “doorbell” or a “phoneringing.”

When a dynamic event trigger 706 is detected, at step 1022 the commandgenerator 502 sends a command specified by the dynamic events 902 in asimilar way as for the command specified by command schedule 901.

The process flow shown in FIG. 10 and described above is an exampleoperational representation of the process executed by the processor 501and is not meant to imply any particular programming techniques,hardware structure, or code structure. For example, programmingtechniques that do not use mode-switched loops and/or event-drivenarchitecture are meant to fall within the ambit of the disclosure.

FIG. 11 shows an example process for verifying execution of a commandsent to a user entertainment device 306 by the command generator 502.Such a process may be used, for example, by the command generator 502 inorder to tune a television 307 to a particular source (e.g., the “Turnon TV to NBC” example of FIG. 6). The process of FIG. 11 is exemplary,and other processes could be used by the command generator in order tocarry out other types of commands.

Prior to block 1101, the command generator 502 generates and transmits(e.g., via IR transmitter 510 b) a command to set the user entertainmentdevice to a selected media content. For example, the command generator502 may formulate one or more IR commands to effect a media sourcechange (e.g., a channel change) or a media input change using a formatrecognized by a television 307. The command generator 502 may access adevice profile 513 that specifies how commands for a particular deviceshould be formatted. Additionally or alternatively, the commandgenerator 502 may access source maps 514 that indicate what sourcescorrespond to particular content.

After sending the command, the processor 501 and/or command generator502 may be unaware of whether the command was successfully executed bythe instructed user entertainment device 306. Many user entertainmentdevices 306 use one-way communication links such as infrared. Thesedevices 306 often lack a back channel to indicate what state the device306 is in so that successful execution of a command can be verified. Forexample, an infrared command sent to a television 307 might fail becausethe television is off, the television is set to the wrong input, orsomething blocked the line of sight between the IR transmitter 510 b andthe television 307.

At block 1101, the command generator 502 may instruct the audio analyzer503 to analyze an audio signal received after the command was sent inorder to determine whether the command was successfully executed. Forexample, the command generator 502 may instruct the audio analyzer 503to analyze an audio signal from a listening device 305 in the same roomas the instructed user entertainment device 306. The audio analyzer 503may analyze the signal as described at step 1006 in order to determineproperties of the audio signal.

At decision 1102, based on the analysis of the audio signal at block1101, the command generator 502 determines whether a class of the audiosignal matches the transmitted command. For example, a class property of“music” or “speech” may indicate that a command to turn on a televisionwas successfully executed (e.g., because sounds from a television oftencontain music or speech). The command generator 502 may use commandpreferences 904, which may map classes to particular commands, in orderto detect which classes indicate successful execution of a particularcommand or commands.

Additionally or alternatively, at decision 1102 the command generator502 may detect a match of one or more other properties in addition to oras an alternative to a class property. For example, the commandgenerator 502 may detect a match of an event with the same room propertyas the device to which the command was sent. The command generator 502may detect a match when a volume property is above a certain threshold.

If no match is detected, at step 1103 the command generator 502 maydetermine that the instructed device 306 is off, set to the wrong input,muted, or has its volume turned down. Accordingly, the command generator502 may send one or more commands to turn the device on, switch theinput, turn up the volume, and/or turn off the mute before looping backto the start of the process. The command generator 502 may randomlychoose one of the options on each successive iteration of the loop, orthe command generator 502 may cycle through each option on successiveiterations.

When a match is detected, the command generator 502 may proceed todecision 1104 to ensure the volume is in an appropriate range. When thevolume property of the detected audio is above or below certainthresholds, the command generator 502 may send one or more volume downor up commands respectively in step 1105. In step 1106, the commandgenerator 502 may then cause the audio analyzer 503 to analyze an audiosignal received after the volume commands were transmitted beforelooping back to decision 1104. When the audio signal is within thevolume thresholds, the process successfully completes.

The monitoring and security system 300 may monitor video events detectedfrom video signals received by video cameras 310. The video analyzer 505may detect video events based on changes in video signals, such asbrightness, color, etc., in similar ways as described for decision 1005.The video camera 310 and/or the video analyzer 505 may detect videoevents when a brightness, color, or other aspect of a video signalreceived by a video camera 310 changes by a threshold amount. In thisway, for example, a video camera 310 pointed at a television or displaydevice may recognize when the television or display device is turned on(e.g., based on the change in brightness, color, etc. associated with adisplayed image). The video analyzer 505 may use classification, imageanalysis, and/or machine vision techniques to categorize such eventsbased on the video signals, and store the analyzed events in the log 601according to the process shown at FIG. 10. For example, the videoanalyzer 505 may detect that lights were turned on or off based onchanges in brightness of a video signal recorded by video camera 310.The video analyzer 505 may detect that a television is tuned to aparticular network and/or channel based on receiving video signalscontaining a network logo or other image element associated with anetwork and/or channel. The video analyzer 505 may detect that aparticular program is being displayed based on recognizing imagesassociated with the program (e.g., using a fingerprinting process tocompare an image fingerprint to a database of image fingerprintsassociated with particular programs). Indications of correspondingevents (e.g., “light on” or “light off” events, events indicating aparticular network, channel, and/or program is being displayed, etc.)may be stored in the logs 601, and the profile generator 504 may furtherdetect groups 602 of events and habits 603 therefrom. The profilegenerator 504 may map such habits to commands for turning lights on andoff, turning a television on and off, tuning to a particular channel,network, or program, and the like according to a command schedule 901 ordynamic events 902. Other monitoring devices that monitor a premises 302may also be used.

The premises controller 301 may detect an event and/or determineinformation about an event by requesting information from a serverassociated with a user entertainment device 306. For example, a serverat the local office 103 (e.g., app server 107) may offer an applicationprogramming interface (API) for accessing information about set top box113. Accordingly, the premises controller 301 may request informationabout an event associated with the set top box 113 and/or a television307 connected to the set top box 113 by sending an API request to theapp server 107. The app server 107 may respond with an indication of achannel, network, and/or program currently tuned by the set top box 113,an on/off status of the set top box 113, and the like. Accordingly, thepremises controller 301 may detect events and/or obtain additionalinformation about detected and/or analyzed events by sending a requestto a device such as an API server. The information obtained from thedevice may be stored in the logs 601, and the profile generator 504 mayfurther detect groups 602 of events and habits 603 therefrom. Theprofile generator 504 may map such habits 603 to commands such asturning a television on and off, tuning to a particular channel,network, or program, and the like, according to a command schedule 901or dynamic events 902.

The monitoring and security system 300 also monitors home automationevents, which may be detected by the automation controller 507. Forexample, the automation controller 507 may communicate with one or moresmart devices such as lights 308 and other smart devices such as smartdoorbells, smart locks, smart appliances, etc. The automation controller507, using home automation protocols or other network connectivity, mayregularly poll connected smart devices to detect whether their state haschanged, and detect events therefrom. For example, the automationcontroller 507 may detect an event whenever an appliance is turned on oroff and/or otherwise changes state. Properties of an event detected bythe automation controller 507 may include an on/off state of the device,a room of the device, an intensity of a light 308, and the like. Thesmart devices may send notifications of their current state and/orchanged state to the automation controller 507.

Furthermore, the command generator 502 and/or automation controller 507may be configured to send commands to smart devices including lights308. The command schedule 901 or dynamic events 902 may contain commandsbased on detected events received from the automation controller 507.When a command is directed to a smart device, the command may be send bythe command generator 502, the automation controller 507, and/or someother component of the premises controller 301 or network controller303.

The premises controller 301 may implement a “welcome” mode after a modeis changed from “away” to “stay” or the monitoring and security system300 otherwise detects that a user has returned to a premises. Accordingto a “welcome” mode, the premises controller 301 may be configured tocontinue simulating a user's presence (e.g., using the command schedule901 and/or dynamic events 902) for a limited time after the monitoringand security system 300 is switched out of an “away” mode. For example,for a five-minute period after switching out of an “away” mode, thepremises controller 301 may continue to transmit commands according to acommand schedule 901 and/or dynamic events 902, thus activating userentertainment devices 306 and/or other devices for a user who has justreturned to the premises. After the expiration of the limited time, thepremises controller 301 may continue operating according to theprocedures associated with the “stay” mode as described herein.

One or more aspects of the disclosure may be embodied in computer-usabledata and/or computer-executable instructions, such as in one or moreprogram modules, executed by one or more computers or other devices.Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types when executed by a processor ina computer or other data processing device. The computer executableinstructions may be stored on one or more computer readable media suchas a hard disk, optical disk, removable storage media, solid statememory, RAM, etc. The functionality of the program modules may becombined or distributed as desired in various embodiments. In addition,the functionality may be embodied in whole or in part in firmware orhardware equivalents such as integrated circuits, field programmablegate arrays (FPGA), application-specific integrated circuits (ASIC), andthe like. Particular data structures may be used to more effectivelyimplement one or more aspects of the invention, and such data structuresare contemplated within the scope of computer executable instructionsand computer-usable data described herein.

Aspects of the disclosure have been described in terms of examples.While exemplary systems and methods as described herein embodyingvarious aspects of the present disclosure are shown, it will beunderstood by those skilled in the art, that the disclosure is notlimited to these embodiments. Modifications may be made by those skilledin the art, particularly in light of the foregoing teachings. Forexample, each of the features of the aforementioned examples may beutilized alone or in combination or sub-combination with elements of theother examples. For example, any of the above described systems andmethods or parts thereof may be combined with the other methods andsystems or parts thereof described above. For example, the steps shownin the figures may be performed in other than the recited order, and oneor more steps shown may be optional in accordance with aspects of thedisclosure. It will also be appreciated and understood thatmodifications may be made without departing from the true spirit andscope of the present disclosure. The description is thus to be regardedas illustrative instead of restrictive on the present disclosure.

What is claimed is:
 1. A method comprising: receiving, by a computingdevice at a premises and while a premises security system is in a firstmode, audio data comprising audio emitted by one or more media devices;determining, based on the audio data, a usage pattern of the one or moremedia devices; and controlling, based on a determination that thepremises security system is in a second mode, the one or more mediadevices to simulate the usage pattern.
 2. The method of claim 1, whereindetermining the usage pattern of the one or more media devicescomprises: detecting, based on the audio data, a plurality of events;and determining one or more common properties of at least two of theplurality of events.
 3. The method of claim 2, further comprisingdetermining properties of each of the plurality of events based onanalyzing the audio data.
 4. The method of claim 1, wherein the audiodata is received from a plurality of listening devices, furthercomprising: causing the plurality of listening devices to record theaudio emitted by the one or more media devices.
 5. The method of claim1, wherein determining the usage pattern comprises classifying eventsdetected in the audio data.
 6. The method of claim 5, whereindetermining the usage pattern further comprises estimating rooms inwhich the events occurred.
 7. The method of claim 1, wherein determiningthe usage pattern further comprises detecting at least two pairs ofevents, wherein a first event of each pair of the at least two pairs isassociated with a first classification and a second event of each pairof the at least two pairs is associated with a second classification. 8.The method of claim 1, wherein controlling the one or more media devicescomprises sending at least one command to the one or more media devices,and further comprising: determining, based on additional audio datareceived after the audio data, whether the at least one command wassuccessfully executed by the one or more media devices.
 9. The method ofclaim 8, wherein determining whether the at least one command wassuccessfully executed comprises determining whether the additional audiodata indicates successful execution of the command.
 10. The method ofclaim 8, wherein determining whether the at least one command wassuccessfully executed comprises detecting whether a volume associatedwith the additional audio data satisfies a threshold.
 11. The method ofclaim 1, wherein the second mode is an armed mode indicating thatresidents of the premises are not within the premises.
 12. A systemcomprising: a premises controller and a media device, wherein thepremises controller comprises: one or more first processors; and memorystoring first computer-readable instructions that, when executed by theone or more first processors, cause the premises controller to: receive,while a premises security system is in a first mode, audio datacomprising audio emitted by one or more media devices; determine, basedon the audio data, a usage pattern of the one or more media devices; andcontrol, based on a determination that the premises security system isin a second mode, the one or more media devices to simulate the usagepattern; wherein the media device comprises: one or more secondprocessors; and memory storing second computer-readable instructionsthat, when executed by the one or more second processors, cause themedia device to receive a command from the premises controller.
 13. Thesystem of claim 12, wherein the first computer-readable instructions,when executed by the one or more first processors, further cause thepremises controller to: detect, based on the audio data, a plurality ofevents; and determine one or more common properties of at least two ofthe plurality of events.
 14. The system of claim 13, wherein the firstcomputer-readable instructions, when executed by the one or more firstprocessors, further cause the premises controller to determineproperties of each of the plurality of events based on analyzing theaudio data.
 15. A premises controller comprising: one or moreprocessors; and memory storing computer-readable instructions that, whenexecuted by the one or more processors, cause the premises controllerto: receive, while a premises security system is in a first mode, audiodata comprising audio emitted by one or more media devices; determine,based on the audio data, a usage pattern of the one or more mediadevices; and control, based on a determination that the premisessecurity system is in a second mode, the one or more media devices tosimulate the usage pattern.
 16. The premises controller of claim 15,wherein the computer-readable instructions, when executed by the one ormore processors, further cause the premises controller to: detect, basedon the audio data, a plurality of events; and determine one or morecommon properties of at least two of the plurality of events.
 17. Thepremises controller of claim 16, wherein the computer-readableinstructions, when executed by the one or more processors, further causethe premises controller to determine properties of each of the pluralityof events based on analyzing the audio data.
 18. The premises controllerof claim 15, wherein the computer-readable instructions, when executedby the one or more processors, cause the premises controller to receivethe audio data by receiving the audio data from a plurality of listeningdevices, and wherein the computer-readable instructions, when executedby the one or more processors, cause the premises controller to causethe plurality of listening devices to record the audio emitted by theone or more media devices.
 19. The premises controller of claim 15,wherein the computer-readable instructions, when executed by the one ormore processors, cause the premises controller to determine the usagepattern by classifying events detected in the audio data.
 20. Thepremises controller of claim 19, wherein the computer-readableinstructions, when executed by the one or more processors, cause thepremises controller to determine the usage pattern by estimating roomsin which the events occurred.